CokeBERT: Contextual knowledge selection and embedding towards enhanced pre-trained language models

Several recent efforts have been devoted to enhancing pre-trained language models (PLMs) by utilizing extra heterogeneous knowledge in graphs (KGs) and achieved consistent improvements on various knowledge-driven NLP tasks. However, most of these knowledge-enhanced PLMs embed static sub-graphs KGs ("knowledge context"), regardless that the required may change dynamically according specific text ("textual context"). In this paper, we propose a novel framework named Coke select contextual context textual for PLMs, which can avoid effect redundant ambiguous cannot match input text. Our experimental results show outperforms baselines typical tasks, indicating effectiveness dynamic understanding. Besides performance improvements, selected describe semantics text-related more interpretable form than conventional PLMs. source code datasets will be available provide details Coke.